GH-36889: [C++][Python] Fix duplicate CSV header when first batch is empty #48718
+76
−3
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ch is empty When writing CSV, if the first record batch was empty, the header would be written twice. This happened because: 1. Header is written to data_buffer_ and flushed during initialization 2. TranslateMinimalBatch returns early for empty batches without modifying data_buffer_ 3. The loop then writes data_buffer_ which still contains the header The fix clears the buffer (resize to 0) when encountering an empty batch, so the subsequent write outputs nothing. Added C++ and Python tests for empty batches at start and in middle of tables. Claude-Generated-By: Claude Code (cli/claude-opus-4-5=1%) Claude-Steers: 2 Claude-Permission-Prompts: 2 Claude-Escapes: 1
|
|
github-actions
bot
added
Component: C++
Component: Python
awaiting review
Signed-off-by: Ruiyang Wang <[email protected]>
Author
|
@wgtmac Hi would you mind to also review this? Thanks! |
HuaHuaY
approved these changes
Jan 9, 2026
Contributor
|
I think we can clear |
Addresses review feedback from HuaHuaY: Instead of clearing the buffer in TranslateMinimalBatch for empty batches, use a WriteAndClearBuffer() helper that writes and clears the buffer in all write paths. This is cleaner because: - Every write follows the same pattern (write -> clear) - Easier to reason about for future write stages - The invariant is explicit: buffer is always clean after flush
Author
|
Clearing after |
Contributor
|
|
wgtmac
reviewed
Jan 13, 2026
github-actions
bot
added
awaiting committer review
Co-authored-by: Gang Wu <[email protected]>
… tests - Rename WriteAndClearBuffer to FlushToSink (shorter, clearer intent) - Consolidate Python tests into a single parameterized test with 3 cases: empty batch at beginning, middle, and end
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Rationale for this change
Fixes #36889
When writing CSV from a table where the first batch is empty, the header gets written twice:
What changes are included in this PR?
The bug happens because:
data_buffer_and flushed duringCSVWriterImplinitializationTranslateMinimalBatchreturns early without modifyingdata_buffer_data_buffer_which still contains stale contentThe fix introduces a
WriteAndClearBuffer()helper that writes the buffer to sink and clears it. This helper is used in all write paths:WriteHeader()WriteRecordBatch()WriteTable()This ensures the buffer is always clean after any flush, making it impossible for stale content to be written again.
Are these changes tested?
Yes. Added C++ tests in
writer_test.ccand Python tests intest_csv.py:Are there any user-facing changes?
No API changes. This is a bug fix that prevents duplicate headers when writing CSV from tables with empty batches.